Material for : Batched Bandit Problems

Motivated by practical applications, chiefly clinical trials, we study the regret achievable for stochastic bandits under the constraint that the employed policy must split trials into a small number of batches. We propose a simple policy, and show that a very small number of batches gives close to minimax optimal regret bounds. As a byproduct, we derive optimal policies with low switching cost for stochastic bandits. In this supplementary material we compare, in simulations, the various policies (grids) introduced in [PRCS15]. These are also compared withUcb2 [ACBF02], which, as noted in [PRCS15], can be seen as an M batch trial with M = Θ(log T ). The simulations are based both on data drawn from standard distributions, and from a real medical trial: specifically, data from Project AWARE, an intervention that sought to reduce the rate of sexually transmitted infections (STI) among high-risk individuals [MFGea13]. Of the three policies introduced here, the minimax grid often does the best at minimizing regret. While all three policies are often bested by Ucb2, it is important to note that the latter algorithm uses an order of magnitude more batches. This makes using Ucb2 for medical trials functionally impossible. For example, in the real data we examine, the data on STI status was not reliably available until at least six months after the intervention. Thus, a three-batch trial would take 1.5 years to run—as intervention and data collection would need to take place three times, six months apart. However, in contrast, Ucb2 would use as many as 56 batches—meaning the overall experiment would take at least 28 years. Despite this extreme difference Supported by ANR grant ANR-13-JS01-0004. Supported by NSF grants DMS-1317308, CAREER-DMS-1053987. Supported by NSF grant SES-1156154. AMS 2000 subject classifications: Primary 62L05; secondary 62C20